Papers with video question answering
Revealing Single Frame Bias for Video-and-Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for video-and-language learning use multiple frames as inputs. |
| Approach: | They propose to use single-frame models for video-and-language learning to investigate temporality in video- and language tasks. |
| Outcome: | The proposed model does not take into account temporal information on video-and-language tasks. |
Do Video Language Models really understand the video contexts? (2025.naacl-srw)
Copied to clipboard
| Challenge: | Recent advances in VideoQA performance have shown that visual language models are effective but the processes of understanding and reasoning in VLMs remain under-explored. |
| Approach: | They propose a framework that incorporates a fine-grained question generation and answering process to measure how well VLMs understand video question answering tasks. |
| Outcome: | The proposed framework incorporates a fine-grained question generation and answering process to measure how well the responses generated by VLMs align with what the model understands. |
MovieCORE: COgnitive REasoning in Movies (2025.emnlp-main)
Copied to clipboard
Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh, Ying Cheng, Hung-Ting Su, Yung-Hao Tang, Shang-Hong Lai, Winston H. Hsu
| Challenge: | MovieCORE is a video question answering dataset that focuses on surface-level comprehension. |
| Approach: | They propose a video question-answer dataset that uses large language models as thought agents to generate and refine high-quality question-anchor pairs. |
| Outcome: | The proposed model improves model reasoning capabilities post-training by 25% . the proposed model is based on a large language model and is scalable to a wide range of tasks . |
Enhancing Temporal Modeling of Video LLMs via Time Gating (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Video Large Language Models neglect temporal information in video data, leading to struggles with temporal-aware video understanding. |
| Approach: | They propose a Time Gating Video LLM (TG-Vid) that employs a time gating module to enhance temporal modeling. |
| Outcome: | The proposed model outperforms existing Large Language Models on video-and-language tasks and ablation studies show that the model outpersforms the existing models. |
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train models on unlabeled web videos are noisy and temporally misaligned . authors propose a method that adds captions and constrained attention loss to improve performance . |
| Approach: | They propose a method that adds captions from video frames as auxiliary text input to provide visual cues for learning better video and language associations. |
| Outcome: | The proposed method outperforms state-of-the-art methods on video-and-language tasks . it adds captions and constrained attention loss to improve model performance . |
CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions (2022.findings-acl)
Copied to clipboard
Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, Deniz Yuret
| Challenge: | Existing models with similar physical and causal understanding capabilities are still underdeveloped. |
| Approach: | They propose a video question answering dataset that requires causal reasoning about physical forces and object interactions. |
| Outcome: | The proposed dataset requires causal reasoning about physical forces and object interactions. |
FIBER: Fill-in-the-Blanks as a Challenging Video Understanding Evaluation Framework (2022.acl-long)
Copied to clipboard
Santiago Castro, Ruoyao Wang, Pingxuan Huang, Ian Stewart, Oana Ignat, Nan Liu, Jonathan Stroud, Rada Mihalcea
| Challenge: | Existing video understanding evaluation frameworks that use fill-in-the-blanks do not reflect real-world tasks. |
| Approach: | They propose to use fill-in-the-blanks as a video understanding evaluation framework and introduce a novel dataset that collects multiple perspectives on the same video. |
| Outcome: | The proposed framework does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation; (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. |
Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing video QA models lack the capacity for deep video understanding and flexible multistep reasoning. |
| Approach: | They propose a video question answering model which performs dynamic multistep reasoning between questions and videos. |
| Outcome: | The proposed model improves on three widely used video QA datasets and displays better interpretability by backtracing along with the attention mechanisms to the video scene graphs. |
Modality Alignment between Deep Representations for Effective Video-and-Language Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing Video-and-Language models do not take into account the different characteristics of video and text representations. |
| Approach: | They propose a method that exploits Centered Kernel Alignment (CKA) to enhance cross-modality attention by combining multiple modalities. |
| Outcome: | The proposed method outperforms conventional multi-modal methods significantly on video QA tasks with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. |
Encoding and Controlling Global Semantics for Long-form Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to find answers for long videos fail to reason over the whole sequence of video, leading to sub-optimal performance. |
| Approach: | They propose a state space layer to integrate global semantics into video . they use a gating unit to enable controllability over the flow of global semantic into visual representations. |
| Outcome: | The proposed framework is able to integrate global semantics into visual representations. |
Video Question Answering: Datasets, Algorithms and Challenges (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in video question answering have led to a surge in popularity . despite the popularity, VideoQA remains one of the greatest challenges . |
| Approach: | They categorize the video question-answer datasets into normal VideoQA, multi-modal VideoQA and knowledge-based VideoQA according to the modalities invoked in the question-announcement pairs. |
| Outcome: | The proposed methods are mainly designed for Factoid QA and inference VideoQA . the proposed methods have been compared with other methods and are robust and interpretable. |
In-the-Wild Video Question Answering (2022.coling-1)
Copied to clipboard
| Challenge: | Existing video understanding datasets focus on human interactions with little attention being paid to the “in the wild” settings. |
| Approach: | They propose a video understanding dataset of videos recorded outdoors . they propose identifying visual support for a given question and answer . |
| Outcome: | The proposed dataset examines the ability of models to understand videos, including video question answering, video captioning, and fill-inthe-blank tasks. |
LifeQA: A Real-life Dataset for Video Question Answering (2020.lrec-1)
Copied to clipboard
Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang, Jia Deng, Rada Mihalcea
| Challenge: | Existing video question answering datasets consist of movies and TV shows, but they are not representative of our day-to-day lives. |
| Approach: | They propose a benchmark dataset for video question answering that focuses on day-to-day situations. |
| Outcome: | The proposed dataset analyzes the challenging but realistic aspects of LifeQA . it consists of video clips and over 2.3k multiple-choice questions . |
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale video-language pre-trained models have shown appealing performance on downstream tasks. |
| Approach: | They propose a video-text model that adapts a pre-trained image-language model into a text-based model without heavy pre-training. |
| Outcome: | The proposed model outperforms existing models on video-text retrieval and video question answering tasks without heavy pre-training. |
Question-Instructed Visual Descriptions for Zero-Shot Video Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for video QA rely on complex architectures, expensive pipelines or closed models like GPTs. |
| Approach: | They propose a single instruction-aware open vision-language model to tackle videoQA using frame descriptions. |
| Outcome: | The proposed framework achieves higher performance than current state-of-the-art models on videoQA benchmarks. |
AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding (2024.findings-emnlp)
Copied to clipboard
Alessandro Suglia, Claudio Greco, Katie Baker, Jose Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, Oliver Lemon
| Challenge: | Current Vision-Language Models (VLMs) focus on third-person view videos, neglecting the richness of egocentric perceptual experience. |
| Approach: | They propose to use the Egocentric Video Understanding Dataset (EVUD) to train VLMs on video captioning and question answering tasks specific to egocentric videos. |
| Outcome: | The proposed model outperforms open-source models including strong Socratic models using GPT-4 as a planner by 3.6% and outperformed Claude 3 and Gemini Pro Vision 1.0. |
TutorialVQA: Question Answering Dataset for Tutorial Videos (2020.lrec-1)
Copied to clipboard
| Challenge: | a new question answering task on instructional videos is needed due to their verbose nature . factoid questions are only a small part of what people actually want to ask on video contents . |
| Approach: | They propose a question answering task on instructional videos based on video transcripts . they use a dataset consisting of 6,000 manually collected triples of (video, question, answer span) |
| Outcome: | The proposed task focuses on screencast tutorial videos pertaining to an image editing program. |
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) are used for video quality assessment, image captioning and video analysis. |
| Approach: | They propose a benchmark to evaluate MLLMs on AIGC videos using coherence validation, error awareness, error type detection and reasoning evaluation tasks. |
| Outcome: | The proposed benchmark evaluates 13 frontier MLLMs on AIGC videos. |
Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining. |
| Approach: | They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts. |
| Outcome: | The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings . |
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding (2026.acl-long)
Copied to clipboard
Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
| Challenge: | Existing methods for MLLMs struggle with fine-grained temporal reasoning . despite advances in video understanding, current methods struggle with time-sensitive tasks . |
| Approach: | They propose a time-stamp-aware multi-segment grounding method that enhances temporal understanding by introducing timestamps. |
| Outcome: | The proposed method outperforms existing methods on time-sensitive tasks and generalizes well across diverse temporal understanding scenarios. |